Generative vs. Discriminative AI
submitted by /u/Neurosymbolic
[link] [comments]
( 9
min )
-
Open
-
Open
-
Open
-
Open
-
Open
Learn your fruits and vegetables
Thanks to DALL-E3 generated educational material, we can bypass the need for teachers and textbook writers.
Can it do fruits? Of course it does fruits!
Or perhaps you would like to learn your berries?
Perhaps you would like to learn your berries in SWEDISH?
I'm learning so much.
( 2
min )
Bonus: learning about fruits and vegetables!
AI Weirdness: the strange side of machine learning
( 2
min )
-
Open
Bayesian Methods for Media Mix Modelling with shape and funnel effects. (arXiv:2311.05587v1 [cs.LG])
In recent years, significant progress in generative AI has highlighted the
important role of physics-inspired models that utilize advanced mathematical
concepts based on fundamental physics principles to enhance artificial
intelligence capabilities. Among these models, those based on diffusion
equations have greatly improved image quality. This study aims to explore the
potential uses of Maxwell-Boltzmann equation, which forms the basis of the
kinetic theory of gases, and the Michaelis-Menten model in Marketing Mix
Modelling (MMM) applications. We propose incorporating these equations into
Hierarchical Bayesian models to analyse consumer behaviour in the context of
advertising. These equation sets excel in accurately describing the random
dynamics in complex systems like social interactions and consumer-advertising
interactions.
( 2
min )
AdjointBackMapV2: Precise Reconstruction of Arbitrary CNN Unit's Activation via Adjoint Operators. (arXiv:2110.01736v2 [cs.LG] UPDATED)
Adjoint operators have been found to be effective in the exploration of CNN's
inner workings [1]. However, the previous no-bias assumption restricted its
generalization. We overcome the restriction via embedding input images into an
extended normed space that includes bias in all CNN layers as part of the
extended space and propose an adjoint-operator-based algorithm that maps
high-level weights back to the extended input space for reconstructing an
effective hypersurface. Such hypersurface can be computed for an arbitrary unit
in the CNN, and we prove that this reconstructed hypersurface, when multiplied
by the original input (through an inner product), will precisely replicate the
output value of each unit. We show experimental results based on the CIFAR-10
and CIFAR-100 data sets where the proposed approach achieves near 0 activation
value reconstruction error.
( 2
min )
Straggler-Resilient Differentially-Private Decentralized Learning. (arXiv:2212.03080v2 [cs.LG] UPDATED)
We consider the straggler problem in decentralized learning over a logical
ring while preserving user data privacy. Especially, we extend the recently
proposed framework of differential privacy (DP) amplification by
decentralization by Cyffers and Bellet to include overall training
latency--comprising both computation and communication latency. Analytical
results on both the convergence speed and the DP level are derived for both a
skipping scheme (which ignores the stragglers after a timeout) and a baseline
scheme that waits for each node to finish before the training continues. A
trade-off between overall training latency, accuracy, and privacy,
parameterized by the timeout of the skipping scheme, is identified and
empirically validated for logistic regression on a real-world dataset and for
image classification using the MNIST and CIFAR-10 datasets.
( 2
min )
Optimization on Manifolds via Graph Gaussian Processes. (arXiv:2210.10962v3 [stat.ML] UPDATED)
This paper integrates manifold learning techniques within a \emph{Gaussian
process upper confidence bound} algorithm to optimize an objective function on
a manifold. Our approach is motivated by applications where a full
representation of the manifold is not available and querying the objective is
expensive. We rely on a point cloud of manifold samples to define a graph
Gaussian process surrogate model for the objective. Query points are
sequentially chosen using the posterior distribution of the surrogate model
given all previous queries. We establish regret bounds in terms of the number
of queries and the size of the point cloud. Several numerical examples
complement the theory and illustrate the performance of our method.
( 2
min )
On neural and dimensional collapse in supervised and unsupervised contrastive learning with hard negative sampling. (arXiv:2311.05139v1 [cs.LG])
For a widely-studied data model and general loss and sample-hardening
functions we prove that the Supervised Contrastive Learning (SCL), Hard-SCL
(HSCL), and Unsupervised Contrastive Learning (UCL) risks are minimized by
representations that exhibit Neural Collapse (NC), i.e., the class means form
an Equianglular Tight Frame (ETF) and data from the same class are mapped to
the same representation. We also prove that for any representation mapping, the
HSCL and Hard-UCL (HUCL) risks are lower bounded by the corresponding SCL and
UCL risks. Although the optimality of ETF is known for SCL, albeit only for
InfoNCE loss, its optimality for HSCL and UCL under general loss and hardening
functions is novel. Moreover, our proofs are much simpler, compact, and
transparent. We empirically demonstrate, for the first time, that ADAM
optimization of HSCL and HUCL risks with random initialization and suitable
hardness levels can indeed converge to the NC geometry if we incorporate
unit-ball or unit-sphere feature normalization. Without incorporating hard
negatives or feature normalization, however, the representations learned via
ADAM suffer from dimensional collapse (DC) and fail to attain the NC geometry.
( 2
min )
Perfectly Accurate Membership Inference by a Dishonest Central Server in Federated Learning. (arXiv:2203.16463v2 [cs.LG] UPDATED)
Federated Learning is expected to provide strong privacy guarantees, as only
gradients or model parameters but no plain text training data is ever exchanged
either between the clients or between the clients and the central server. In
this paper, we challenge this claim by introducing a simple but still very
effective membership inference attack algorithm, which relies only on a single
training step. In contrast to the popular honest-but-curious model, we
investigate a framework with a dishonest central server. Our strategy is
applicable to models with ReLU activations and uses the properties of this
activation function to achieve perfect accuracy. Empirical evaluation on visual
classification tasks with MNIST, CIFAR10, CIFAR100 and CelebA datasets show
that our method provides perfect accuracy in identifying one sample in a
training set with thousands of samples. Occasional failures of our method lead
us to discover duplicate images in the CIFAR100 and CelebA datasets.
( 2
min )
GAN-based Tabular Data Generator for Constructing Synopsis in Approximate Query Processing: Challenges and Solutions. (arXiv:2212.09015v2 [cs.DB] UPDATED)
In data-driven systems, data exploration is imperative for making real-time
decisions. However, big data is stored in massive databases that are difficult
to retrieve. Approximate Query Processing (AQP) is a technique for providing
approximate answers to aggregate queries based on a summary of the data
(synopsis) that closely replicates the behavior of the actual data, which can
be useful where an approximate answer to the queries would be acceptable in a
fraction of the real execution time. This study explores the novel utilization
of Generative Adversarial Networks (GANs) in the generation of tabular data
that can be employed in AQP for synopsis construction. We thoroughly
investigate the unique challenges posed by the synopsis construction process,
including maintaining data distribution characteristics, handling bounded
continuous and categorical data, and preserving semantic relationships and then
introduce the advancement of tabular GAN architectures that overcome these
challenges. Furthermore, we propose and validate a suite of statistical metrics
tailored for assessing the reliability of the GAN-generated synopses. Our
findings demonstrate that advanced GAN variations exhibit a promising capacity
to generate high-fidelity synopses, potentially transforming the efficiency and
effectiveness of AQP in data-driven systems.
( 2
min )
Antenna Response Consistency Driven Self-supervised Learning for WIFI-based Human Activity Recognition. (arXiv:2310.06328v2 [cs.LG] UPDATED)
Self-supervised learning (SSL) for WiFi-based human activity recognition
(HAR) holds great promise due to its ability to address the challenge of
insufficient labeled data. However, directly transplanting SSL algorithms,
especially contrastive learning, originally designed for other domains to CSI
data, often fails to achieve the expected performance. We attribute this issue
to the inappropriate alignment criteria, which disrupt the semantic distance
consistency between the feature space and the input space. To address this
challenge, we introduce \textbf{A}ntenna \textbf{R}esponse \textbf{C}onsistency
(ARC) as a solution to define proper alignment criteria. ARC is designed to
retain semantic information from the input space while introducing robustness
to real-world noise. Moreover, we substantiate the effectiveness of ARC through
a comprehensive set of experiments, demonstrating its capability to enhance the
performance of self-supervised learning for WiFi-based HAR by achieving an
increase of over 5\% in accuracy in most cases and achieving a best accuracy of
94.97\%.
( 2
min )
A Survey of Federated Unlearning: A Taxonomy, Challenges and Future Directions. (arXiv:2310.19218v2 [cs.LG] UPDATED)
With the development of trustworthy Federated Learning (FL), the requirement
of implementing right to be forgotten gives rise to the area of Federated
Unlearning (FU). Comparing to machine unlearning, a major challenge of FU lies
in the decentralized and privacy-preserving nature of FL, in which clients
jointly train a global model without sharing their raw data, making it
substantially more intricate to selectively unlearn specific information. In
that regard, many efforts have been made to tackle the challenges of FU and
have achieved significant progress. In this paper, we present a comprehensive
survey of FU. Specially, we provide the existing algorithms, objectives,
evaluation metrics, and identify some challenges of FU. By reviewing and
comparing some studies, we summarize them into a taxonomy for various schemes,
potential applications and future directions.
( 2
min )
Familiarity-Based Open-Set Recognition Under Adversarial Attacks. (arXiv:2311.05006v1 [cs.CV])
Open-set recognition (OSR), the identification of novel categories, can be a
critical component when deploying classification models in real-world
applications. Recent work has shown that familiarity-based scoring rules such
as the Maximum Softmax Probability (MSP) or the Maximum Logit Score (MLS) are
strong baselines when the closed-set accuracy is high. However, one of the
potential weaknesses of familiarity-based OSR are adversarial attacks. Here, we
present gradient-based adversarial attacks on familiarity scores for both types
of attacks, False Familiarity and False Novelty attacks, and evaluate their
effectiveness in informed and uninformed settings on TinyImageNet.
( 2
min )
Covering Number of Real Algebraic Varieties: Improved Bound and Applications. (arXiv:2311.05116v1 [math.AG])
We prove an upper bound on the covering number of real algebraic varieties,
images of polynomial maps and semialgebraic sets. The bound remarkably improves
the best known bound by Yomdin-Comte, and its proof is much more
straightforward. As a consequence, our result gives a bound on volume of the
tubular neighborhood of a real variety, improving the results by Lotz and
Basu-Lerario. We apply our theory to three main application domains. Firstly,
we derive a near-optimal bound on the covering number of low rank CP tensors.
Secondly, we prove a bound on the sketching dimension for (general) polynomial
optimization problems. Lastly, we deduce generalization error bounds for deep
neural networks with rational or ReLU activations, improving or matching the
best known results in the literature.
( 2
min )
Improving Computational Efficiency for Powered Descent Guidance via Transformer-based Tight Constraint Prediction. (arXiv:2311.05135v1 [math.OC])
In this work, we present Transformer-based Powered Descent Guidance (T-PDG),
a scalable algorithm for reducing the computational complexity of the direct
optimization formulation of the spacecraft powered descent guidance problem.
T-PDG uses data from prior runs of trajectory optimization algorithms to train
a transformer neural network, which accurately predicts the relationship
between problem parameters and the globally optimal solution for the powered
descent guidance problem. The solution is encoded as the set of tight
constraints corresponding to the constrained minimum-cost trajectory and the
optimal final time of landing. By leveraging the attention mechanism of
transformer neural networks, large sequences of time series data can be
accurately predicted when given only the spacecraft state and landing site
parameters. When applied to the real problem of Mars powered descent guidance,
T-PDG reduces the time for computing the 3 degree of freedom fuel-optimal
trajectory, when compared to lossless convexification, from an order of 1-8
seconds to less than 500 milliseconds. A safe and optimal solution is
guaranteed by including a feasibility check in T-PDG before returning the final
trajectory.
( 2
min )
Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels. (arXiv:2311.05265v1 [cs.CL])
In this paper, we address the limitations of the common data annotation and
training methods for objective single-label classification tasks. Typically,
when annotating such tasks annotators are only asked to provide a single label
for each sample and annotator disagreement is discarded when a final hard label
is decided through majority voting. We challenge this traditional approach,
acknowledging that determining the appropriate label can be difficult due to
the ambiguity and lack of context in the data samples. Rather than discarding
the information from such ambiguous annotations, our soft label method makes
use of them for training. Our findings indicate that additional annotator
information, such as confidence, secondary label and disagreement, can be used
to effectively generate soft labels. Training classifiers with these soft
labels then leads to improved performance and calibration on the hard label
test set.
( 2
min )
Detecting Relevant Information in High-Volume Chat Logs: Keyphrase Extraction for Grooming and Drug Dealing Forensic Analysis. (arXiv:2311.04905v1 [cs.CL])
The growing use of digital communication platforms has given rise to various
criminal activities, such as grooming and drug dealing, which pose significant
challenges to law enforcement and forensic experts. This paper presents a
supervised keyphrase extraction approach to detect relevant information in
high-volume chat logs involving grooming and drug dealing for forensic
analysis. The proposed method, JointKPE++, builds upon the JointKPE keyphrase
extractor by employing improvements to handle longer texts effectively. We
evaluate JointKPE++ using BERT-based pre-trained models on grooming and drug
dealing datasets, including BERT, RoBERTa, SpanBERT, and BERTimbau. The results
show significant improvements over traditional approaches and demonstrate the
potential for JointKPE++ to aid forensic experts in efficiently detecting
keyphrases related to criminal activities.
( 2
min )
-
Open
Bayesian sequential design of computer experiments for quantile set inversion. (arXiv:2211.01008v3 [stat.ML] UPDATED)
We consider an unknown multivariate function representing a system-such as a
complex numerical simulator-taking both deterministic and uncertain inputs. Our
objective is to estimate the set of deterministic inputs leading to outputs
whose probability (with respect to the distribution of the uncertain inputs) of
belonging to a given set is less than a given threshold. This problem, which we
call Quantile Set Inversion (QSI), occurs for instance in the context of robust
(reliability-based) optimization problems, when looking for the set of
solutions that satisfy the constraints with sufficiently large probability. To
solve the QSI problem, we propose a Bayesian strategy based on Gaussian process
modeling and the Stepwise Uncertainty Reduction (SUR) principle, to
sequentially choose the points at which the function should be evaluated to
efficiently approximate the set of interest. We illustrate the performance and
interest of the proposed SUR strategy through several numerical experiments.
( 2
min )
Simple steps are all you need: Frank-Wolfe and generalized self-concordant functions. (arXiv:2105.13913v7 [math.OC] UPDATED)
Generalized self-concordance is a key property present in the objective
function of many important learning problems. We establish the convergence rate
of a simple Frank-Wolfe variant that uses the open-loop step size strategy
$\gamma_t = 2/(t+2)$, obtaining a $\mathcal{O}(1/t)$ convergence rate for this
class of functions in terms of primal gap and Frank-Wolfe gap, where $t$ is the
iteration count. This avoids the use of second-order information or the need to
estimate local smoothness parameters of previous work. We also show improved
convergence rates for various common cases, e.g., when the feasible region
under consideration is uniformly convex or polyhedral.
( 2
min )
Optimization on Manifolds via Graph Gaussian Processes. (arXiv:2210.10962v3 [stat.ML] UPDATED)
This paper integrates manifold learning techniques within a \emph{Gaussian
process upper confidence bound} algorithm to optimize an objective function on
a manifold. Our approach is motivated by applications where a full
representation of the manifold is not available and querying the objective is
expensive. We rely on a point cloud of manifold samples to define a graph
Gaussian process surrogate model for the objective. Query points are
sequentially chosen using the posterior distribution of the surrogate model
given all previous queries. We establish regret bounds in terms of the number
of queries and the size of the point cloud. Several numerical examples
complement the theory and illustrate the performance of our method.
( 2
min )
A Generalized Variable Importance Metric and Estimator for Black Box Machine Learning Models. (arXiv:2212.09931v2 [stat.CO] UPDATED)
The aim of this study is to define importance of predictors for black box
machine learning methods, where the prediction function can be complex and
cannot be represented by statistical parameters. In this paper we defined a
``Generalized Variable Importance Metric (GVIM)'' using the true conditional
expectation function for a continuous or a binary response variable. We further
showed that the defined GVIM can be represented as a function of the
Conditional Average Treatment Effect (CATE) for multinomial and continuous
predictors. Then we propose how the metric can be estimated using using any
machine learning models. Finally using simulations we evaluated the properties
of the estimator when estimated from XGBoost, Random Forest and a mis-specified
generalized additive model.
( 2
min )
Uncertainty Wrapper in the medical domain: Establishing transparent uncertainty quantification for opaque machine learning models in practice. (arXiv:2311.05245v1 [cs.LG])
When systems use data-based models that are based on machine learning (ML),
errors in their results cannot be ruled out. This is particularly critical if
it remains unclear to the user how these models arrived at their decisions and
if errors can have safety-relevant consequences, as is often the case in the
medical field. In such cases, the use of dependable methods to quantify the
uncertainty remaining in a result allows the user to make an informed decision
about further usage and draw possible conclusions based on a given result. This
paper demonstrates the applicability and practical utility of the Uncertainty
Wrapper using flow cytometry as an application from the medical field that can
benefit from the use of ML models in conjunction with dependable and
transparent uncertainty quantification.
( 2
min )
